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ABSTRACT 



This paper recounts the process of integrating industry into 
the assessment process in engineering education developed by the Synthesis 
Coalition, a group of colleges and schools working together to improve 
engineering education through development and implementation of curriculum 
reforms. Originating in the Coalition's efforts to introduce mechatronics 
into the curriculum, the process involved use of an advisory industrial 
board, the Mechatronics Industry Board (MIB) , This Board broadened its 
mission to become involved in the entire process of the outcomes based 
assessment program. This paper describes the two track planning process 
developed to capitalize on the different expertise of the faculty and the 
MIB: first, to identify valued student learning outcomes and, second, to 
validate the Synthesis Assessment Framework, a document laying out the 
Coalition's student learning goals, learning outcomes, and criteria for 
measuring these outcomes. Twenty scenarios describing situations which 
engineers commonly encounter were developed and analyzed for necessary skills 
and abilities. The project also developed four assessment tools to measure 
student learning including two self and peer assessment questionnaires 
related to teamwork and oral presentation, a design project report, and a 
scenario assignment for which a scoring rubric was developed by MIB and the 
Coalition. (Contains 14 references.) (DB) 
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A COLLABORATIVE ROLE FOR INDUSTRY IN 
ASSESSING STUDENT LEARNING 

ABSTRACT 

In engineering there is much common ground between the accreditation criteria for the 
Accreditation Board for Engineering and Technology (ABET), industry expectations for the 
preparation of engineers, and how faculty work with students to meet those expectations. This 
paper will describe a process developed and implemented by the Synthesis Coalition to integrate 
industry more fully into the assessment process of the Coalition and its participating schools. 
The Coalition of seven engineering schools (California Polytechnic at San Luis Obispo, Hampton 
University, Iowa State University, Southern University, Stanford, Tuskegee University, and the 
University of California at Berkeley) is a member of the NSF sponsored Engineering Education 
Coalition Program which focuses on improving engineering education through curriculum and 
pedagogical reform efforts. In this model, industry collaborated with faculty and students as full 
partners in designing the assessment process, and testing and evaluating the assessment tools and 
scoring rubrics. 
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A COLLABORATIVE ROLE FOR INDUSTRY IN 
ASSESSING STUDENT LEARNING 



INTRODUCTION 

The importance of assessing student learning outcomes has increased dramatically in the 
accreditation and continuous improvement processes of colleges and universities. Critical to the 
success of these assessment efforts is the need for an institution’s stakeholders to collaborate in 
all aspects of the process, from the initial planning activities to the evaluation and eventual use of 
the assessment results. For the Synthesis Coalition (a member of the NSF sponsored Engineering 
Education Coalition Program to improve engineering education) this meant collaboration among 
students, faculty and administrators from its participating schools (California Polytechnic at San 
Luis Obispo, Hampton University, Iowa State University, Southern University, Stanford, 
Tuskegee University and the University of California at Berkeley) and representatives of the 
industries who ultimately employ their graduates (e.g., Hewlett-Packard, Motorola, and 
Maytag). 

Collaboration between industry and faculty from the institutions participating in the Coalition 
led to a comprehensive assessment approach that encompassed many of the learning outcomes 
required by the Accreditation Board for Engineering and Technology (ABET). There are eleven 
learning outcomes that range from the ability to apply knowledge of math, science and 
engineering to the ability to function on multi-disciplinary teams. The purpose of this paper is to 
describe how the Synthesis Coalition engaged and involved industry collaborators in its 
assessment program. While this paper focuses on the processes described at the Coalition level, 
many of the lessons learned during the process are applicable to the assessment needs of 
departments and colleges. 

BACKGROUND OF INDUSTRY INVOLVEMENT IN ENGINEERING EDUCATION 

There is a long history of industry involvement in higher education, from endowing institutions 
such as MIT and Carnegie Mellon to participating in career fairs, sharing training and research 
facilities, and staffing advisory boards. These collaborative relationships can build communication 
bridges, provide a forum for discussing broad issues affecting both constituents, inform one 
another about future trends and needs, improve the relevance of education, and generate increased 
support from industry. 

Engineering colleges, however, have tended to focus their collaboration efforts with industry 
around research activities, such as building ties through research and development efforts. Prism 
(the journal of the American Society of Engineering Education) provides an excellent illustration 
of this focus. The articles in the Special Industry Issue (Dalheim, 1998) highlight the work of 
several school and industry partnerships to develop university based business incubators, 
strengthen community ties via industry funding for labs, and relax the agreement process between 
faculty and industry around licensing and patents. While these efforts do promote collaboration 
around improving engineering education, they do not involve industry in an active collaboration 
around curriculum reform or student learning. 
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Ironically, Industry has been intimately involved in (re)defining engineering education at the 
national level (e.g., participating in the new ABET accreditation process). And at more the local 
level, individual engineering colleges and schools have created industrial advisory boards to 
provide them with advice on a curriculum's relevance to current industry practices, or give general 
feedback on the strengths and weaknesses of graduating students as they enter industry. Here 
too however, industry advisory boards have been kept to the periphery, used only to react to 
pre-defmed learning goals or simply asked to complete surveys regarding the skills of graduates 
(McMartin and McGourty, 1999). 

Expanding industry’s role to be active collaborators in assessment and accreditation processes 
including, planning, developing, implementing, and evaluating assessment data is important for 
several reasons. First, involving industry at the planning and development stages insures that 
what colleges assess is valued by the campus community and the larger community, i.e., 
employers, regents/trustees, graduate schools, state legislatures, and parents (Davis, 1989; 
Immerwahr, 1999; Lincoln and Guba, 1989; Rao, 1999). Second, collaboration with industry 
strengthens accountability, because all parties build a common understanding of the language and 
meaning of assessment, assessment results, and interpretation of those results (McMartin, Van 
Duzer and Agogino, 1998; Sheppard, Reamon, Friedlander, Kerns, Leifer, Marincovich, and 
Toye, 1998). Last, and particular to the needs of engineering schools and colleges, collaboration 
with industry meets the requirements of ABET which mandates that industry be incorporated 
into their assessment and continuous improvement efforts (ABET, 1997). 

THE SYNTHESIS COALITION APPROACH TO ASSESSMENT 

The Synthesis Coalition colleges and schools are a part of a nation-wide effort to improve 
engineering education promoted by the National Science Foundation Engineering Directorate. The 
primary goals of Synthesis were to develop, institutionalize, assess, and disseminate a set of 
curriculum reforms. Therefore, assessment activities for the Coalition had to meet the needs of 
extremely diverse institutions with regards to institutional size, student population, faculty 
interest, and institutional resources. Coalition faculty were interested in knowing about the 
impact of their individual course and curriculum reforms, but were only marginally intrigued by 
the notion of assessment at the Coalition level. For the most part, they had very little background 
or knowledge about assessment, they were suspicious about its purpose, and saw little relevance 
to it in their daily teaching responsibilities. Thus, they were not highly motivated to participate 
in assessment at either the campus or Coalition level. 

Recognizing the challenges to successfully implementing assessment under these conditions, the 
Synthesis assessment team developed a strategic faculty development program that focused first 
on educating faculty about assessment. At the same time it created tools and processes that 
would meet particular campus needs as well as the Coalition’s. Integral to this process was a 
belief held by the team that the faculty should not shoulder assessment alone. One of the goals 
of Synthesis was to insure that current industrial practices were taught and experienced in the 
classroom, this was a natural opening into which it became possible to bring industry into the 
assessment process. 
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There were multiple purposes behind the efforts to integrate industry into its assessment 
program. First, the Coalition’s members believed that their involvement would vastly improve 
the assessment process by injecting industrial expertise to the program. Emphasizing their role in 
working with faculty and students to identify and articulate student learning outcomes was an 
effective means of insuring the outcomes were those valued by both industry and the academy. 
Another intended result of this joint process was for industry representatives to gain insight into 
the possibilities and limitations of teaching engineering: at the same time faculty (and students) 
would gain a better understanding of the needs, language, and nuances associated with industry's 
view of essential engineering skills. Finally, by increasing the opportunities for dialogue between 
faculty and industry representatives we hoped to raise questions about student learning that 
would motivate the faculty to be more fully involved in the assessment program. 

THE PROCESS FOR INVOLVING INDUSTRY 
The Mechatronics Advisory Board 

Prior to initiating the assessment program, the Coalition had created an industrial board to advise 
it on efforts to introduce Mechatronics into the curriculum. This board, the Mechatronics 
Industry Board (MIB) included representatives from: Berkeley Engineering Systems, Ford 
Motor Co., Hewlett Packard, Maytag, Motorola, Raychem, Rockwell International, and Xerox 
PARC. Because Mechatronics is a relatively new discipline in engineering (Auslander, 1996) the 
Coalition’s management team recruited project managers to be board members because it was felt 
that they would know best the kinds of skills and knowledge needed in this emerging field. 

These selection criteria, originally selected because of the newness of the field, became a critical 
factor in the success of the assessment program. The board members, because they were working 
project managers, were able to articulate what engineers actually do on a daily basis. For example, 
they could describe in detail how engineers worked on teams to solve design problems. The fact 
that the MIB was made up of working engineers rather than industry recruiters or CEOs also lent 
it instant credibility with the faculty. The faculty (many who had come to academe from 
industry) reacted more positively to the project managers who grounded their opinions and 
descriptions of needs in actual practices and engineering problems familiar to the faculty. 

The MIB was a strong addition to the assessment process because of its strong credibility with 
the faculty and its members supported the Coalition’s goals. The MIB was invited to become 
more actively involved in the entire process of the outcomes based assessment program. All of 
the members accepted and continued to serve on the board, even though they were asked to 
increase their commitment to the project (e.g., attend more meetings at their own expense.) 

The Two Track Planning Process 

A two-track assessment planning process (see Figure 1) was developed to capitalize on the 
different expertise of the faculty and the MIB. The first track focused on identifying and agreeing 
on the student leaning outcomes to be measured in the assessment program. Through structured 
conversations, brainstorming, and negotiation among the MIB, faculty, and students, a common 
set of learning objectives emerged that represented their combined interests. The second track 
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focused on validating the outcomes and abilities based on MIB members engineering and project 
management expertise. MIB members developed and described a set of scenarios of a "day in the 
life" of an engineer. These scenarios became windows for faculty to understand more fully the 
challenges engineers encounter in industry, and provided a means for validating the relevance of 
the student learning outcomes to engineering education. 



Track 1: Identification Track 2: Validation 




Figure 1 . Two-Track Assessment Planning Process 

Track One: Identifying Valued Student Learning Outcomes 

Three goals drove the decision to involve industry stakeholders in Track 1 . The goals laid the 
foundation for creating an organizational structure to encourage and support an ongoing 
conversation with faculty. These conversations were designed for all of the stakeholders to: 

• come to consensus on the subset of engineering learning outcomes critical to industry and the 
engineering programs 

• agree on common language to describe those learning outcomes and define criteria for 
measuring them 

• create a sense of ownership in the assessment process that would in turn, energize all the 
participants 

The forum created by the Coalition to encourage this conversation centered on establishing a 
series of small group brainstorming and working sessions between faculty and members of the 
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MIB. The initial sessions, held over the course of a weekend retreat resulted in a set of 
remarkable insights for all the participants. For example, common words such as “teamwork” 
and “communication” turned out to have significant and different meanings for each type of 
stakeholder. And, while there was a significant overlap between the priorities of industry and 
faculty, there were also important differences based on their operating environment, i.e., 
engineers can decide if a project is worth pursuing while students have no such option - they 
must complete assignments. So, by examining one another’s views of university/industry 
practices the participants discovered truly common goals, which in turn, reinforced their 
commitment to developing and implementing effective outcomes assessment. As a result of these 
conversations the group was able to arrive at consensus regarding the outcomes most critical to 
industry and the Coalition. 

Early on in the collaborative process, the stakeholders’ conversations revealed that differences in 
their languages had in fact, facilitated agreement without true understanding. For instance, while 
both faculty and MIB members agreed that communication skills were important, faculty 
interpreted communication skills to mean formal written and oral presentations, while the MIB 
interpreted communication skills to mean interpersonal and informal communication with co- 
workers, clients, and so forth. Thus, the group’s structured conversations revealed sometimes- 
subtle nuances underlying the different stakeholders’ operational definitions. Discussion of these 
nuances led to better understanding on the part of the stakeholders of the abilities students need 
in order to be successful engineers. Building on this more complete understanding of the 
Synthesis learning outcomes, the group negotiated and operationalized the learning abilities, 
pedagogical activities, and measurement criteria associated with the five most valued Synthesis 
learning goals (communication, hands-on learning, teamwork, open-ended problem solving, and 
multi-disciplinary design). Taken together, the outcome of these sessions was a list of 
approximately 500 items. 

This list was culled down to a manageable set of outcomes for assessment. A task force (two 
faculty members, one industry representative and two students) reviewed the list of 500, 
negotiated priorities, and wiimowed it down into a set of approximately 6 outcomes for each goal 
(about 30 outcomes total.) This negotiated set of abilities was sent to all of the stakeholders for 
review and comment. As part of this review, faculty were asked to rate the degree to which their 
Synthesis courses emphasized the development of each of the abilities on the list. These data and 
the comments from the stakeholder review were used to further refine the list of abilities and 
criteria into the Synthesis Assessment Framework {Siynih&sis Coalition, 1997). 

While the common ground between industry and the Synthesis learning goals became the focus of 
the assessment plan, the process also generated an important understanding of the differences in 
values, goals, and environments in the stakeholders’ worlds. For instance, issues regarding the 
nature of ambiguity in engineering were brought out. MIB members felt that students should be 
prepared for work situations where projects are ill defined, there are significant economic, 
technical and sometimes environmental design restraints, and criteria for success are uncertain. 
Faculty described that the difficulties involved in (re)creating such a situation in the classroom, 
noting especially that if course assignments or projects were deliberately ill-defined and lacked 
criteria for success, students could quite legitimately complain about the lack of fair grading 
practices. As a result of this kind of discussion, faculty gained insight into industry’s 
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expectations regarding the skills and abilities engineering graduates need when entering industry, 
and the MIB began to understand the constraints faculty face in teaching. 

The products that resulted from Track 1 centered on a deepened understanding of stakeholders’ 
values regarding engineering education. The main product, a draft of the Synthesis Assessment 
Framework, described those values and specifically defined the set of student learning outcomes 
to be assessed. The second product was a deepened imderstanding on the part of the 
stakeholders of the expectations industry has for engineering education, and the challenge faculty 
face in creating opportunities for that education. By opening up a dialogue aroimd these issues, 
the stakeholders' commitment to the assessment program increased because they now imderstood 
the impact the curriculum reforms could have on student learning. Now they genuinely wanted to 
answer the question: How did Synthesis reforms affect student learning? 

Track Two: Validating Synthesis Assessment Framework 

The draft version of the Synthesis Assessment Framework laid out the Coalition’s student 
learning goals, learning outcomes, and criteria for measuring those outcomes. This draft however, 
was imtested in that it represented only the collective wisdom of the stakeholder group, not 
actual practice. It seemed to represent what a graduating student ought to know and be able to do 
in order to be successful engineers, but it remained unclear that these were in fact outcomes 
engineering foimd necessary. 

In order to test the relationship of the outcomes to actual engineering practice, the MIB 
representatives developed a set of scenarios which described situations engineers commonly 
encoimter. These situations reflected either common or critical events in the daily life of an 
engineer. The MIB representatives wrote approximately 20 different scenarios. Each laid out a 
particular problem or task, provided a broad description of the surroimding context, and 
integrated aspects of the Synthesis learning goals. For example, one scenario described a series of 
problems associated with regulating heat in a Xerox machine. The description included 
information about the machine, client use and feedback, and some of the technical problems 
associated with the problem. Unlike case studies, the scenarios did not always have a solution, 
primarily because some of them had not yet been solved by industry. 

The scenarios, because of their rich description of the context of the situation, the industrial 
environment, and the kind of ‘day in the life” problems faced by engineers were an excellent 
source for examining if the kinds of student learning promoted by Synthesis reforms were used in 
industry. Once they were written and collected, the assessment team conducted a content 
analysis of the scenarios. Skills, abilities, and activities that occurred naturally in the situations 
described in the scenarios were listed and compared with those drawn from the stakeholder 
brainstorming and negotiation process. The analysis revealed a good overlap; approximately 80% 
of the activities and abilities matched. The abilities and activities within the overlap became the 
primary focus of the assessment program. 

Just as the discussions between the stakeholders revealed differences, so to did the analysis of 
the scenarios. Issues such as the ability to supervise others, to talk about what does not exist, 
and to decide if a problem is worth pursuing were all skills central to success in industry, yet 

O 6 




10 



seem to be ill suited to higher education. These differences legitimized for the stakeholders the 
outcomes of engineering education, and highlighted areas for future discussion and consideration. 

Developing and Testing Assessment Tools 

The Coalition developed and designed a set of four assessment tools to measure student learning. 
Two of the tools were self and peer assessment questionnaires related to teamwork and oral 
presentation skills. The other two tools relied upon student written work. The most detailed of 
these two was the design project report. Design project reports are typically required for in 
engineering design courses. Collecting these reports and analyzing their contents could provide 
the Coalition with an in-depth look at student ability across almost all of the selected learning 
outcomes. The final tool was the scenario assignment that was based on the scenarios developed 
by the MIB. These assignments were short problem solving tasks students were asked to 
complete as homework. Each tool and its accompanying scoring rubric was designed by the 
assessment team and underwent a series of pilot tests to determine its dependability. Each tool 
was also reviewed by and tested on the MIB. For the purposes of this paper, the discussion will 
focus on the scenario assignment. 



Scenario Based Assignments 

The scenarios provided a natural link for involving the MIB in tool development and testing. As 
the Coalition’s assessment team reviewed the scenarios during the content analysis, it became 
clear that these documents were potentially rich teaching tools and possible assessment tools. 
Scenarios described actual engineering problems faced in industry. In fact, several of the 
scenarios described problems which industry had not yet solved. This being the case, they could 
easily be used to as the basis for a performance assessment tool. 

The scenarios were re-written as a qualitative assessment tool designed to assess students' 
knowledge of engineering practices, teamwork, and problem solving. The scenarios were also 
performance assessments in that students could demonstrate their ability to solve a project 
management problem. Rather than solving the technical or design problem presented in the 
scenario, students were asked to describe how they would go about finding the solution to the 
technical problem with a team of engineers. Students, when responding to the scenarios took on 
the role of a project manager who must prepare a team to tackle the project while he or she is out 
of town on business. Typically, students completed the scenarios in one to two hours. 

The idea behind the scenario assignments was to provide students with the kinds of tasks that 
they would confront in actual practice in order to engage them in realistic problem solving, and to 
learn how to flexibly apply their knowledge (Rogers, 1994; Yokomoto, Walter and Ware, 1995). 
Specifically, in a scenario assignment an engineering student was asked to apply process-oriented 
knowledge and abilities such as the ability to define the problem and to undertake a plan, which 
will lead to a solution to the problem. Therefore, the scenario assignment was not intended to 
measure a student’s scientific knowledge. Rather, it is a realistic open- ended task that draws on a 
student’s critical thinking skills as well as problem formulation and management expertise. 
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Scenario assignments differ from problem based learning often used in medical education (Brown, 
1996) in that it was not used in a collaborative manner in order to learn specific content material. 
The scenario assignment was also similar to ‘goal-based scenarios’ which are hypothetical cases 
that embody the kinds of problems that professionals may face on a job (Herman, Aschbacher 
and Winters, 1992). Goal-based scenarios have been used by consulting companies for teaching 
and training professional staff. The feature that the scenario assignments, problem-based learning 
and goal-based scenarios have in common was they afford the opportunity for students to apply 
their knowledge in the context of a meaningful realistic problem situation. See Figure 2 for an 
example problem presented in a scenario. 



Modem life seems to revolve around electronic devices, and the trend with these devices has 
always been towards smaller and smaller products. One component, which is critical, but has so 
far confounded many engineers and scientists, is the design of a better battery. 

For this problem, assume that some new materials have been developed which have exciting new 
electro-chemical properties. Your company has decided to manufacture a new battery with them, 
and has asked you to prepare an outline for the design and manufacture of the new battery. 

Since the battery is intended for small devices, such as laptop computers, the size of the battery 
is critical. The output voltage of a battery though, is a function of the number of layers of anode 
and cathode materials and is therefore fixed. 

Figure 2 . Example Scenario 

The Role of MIB in Testing Scenario Based Assignments 

The fact that the MIB had created the scenarios and that they were being turned into assessment 
tools was a major factor in motivating them to stay involved in the assessment program. Writing 
the scenarios had been a time consuming process and once they saw how their work had been 
immediately turned into an assessment tool, they felt their work had been rewarded. They were 
eager to work with a small group of faculty and graduate students to pilot test the tools and to 
develop the scoring rubric. 

To pilot test the scenario assignments, the MIB completed several of the assignments. Once 
they had completed them, they participated in a focus group to discuss the tool. The results of 
the focus group were used to refine the scenario and the directions for completing it. Their 
written responses were also used to help calibrate the scoring rubric after data were collected 
from the students. The MIB also participated in testing the rubric. They scored the set of the 
scenarios they had completed and those completed by Coalition faculty. This exercise gave the 
assessment team valuable feedback regarding the complexity of the rubric. Several of the MIB 
members volunteered to do a written analysis of the tool. Others provided the team with 
examples of personnel evaluation forms used in their companies. These criteria were compared 
to the rubrics to once again check the relevance of the Coalition’s assessment process to those of 
industry. 

Each assessment tool and associated scoring rubric was pilot tested with the MIB in much the 
same manner as the scenario assignment. The MIB members’ response to the pilot tests was 
quite positive - several noted that they had learned a great deal about themselves in relation to 
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the learning outcomes. By involving them in the initial pilot tests they learned about assessment 
in a hands-on manner. As a result, they began to trust that the process would result in 
assessment data that would be meaningful to industry. In fact, the MIB became very strong 
advocates for the Coalition’s assessment plan to campus administrators and to the Coalition’s 
funder, NSF. 

CONCLUSION 

The collaboration process described in this paper seamlessly integrated industry into the 
assessment process. The conversation around engineering education and the process of 
identifying and articulating the set of learning outcomes most valued by the stakeholders brought 
the faculty, students, and industry representatives to a more sophisticated understanding of each 
other's worlds. This understanding created a platform for building assessment on the common 
ground shared by the stakeholders. The end result was a set of explicitly defined learning 
outcomes that were valued by both the academy and its industry partners. The high value placed 
on participation led to a real sense of ownership that encouraged the stakeholders' continued 
engagement in all aspects of the assessment project. 

Ultimately, the process of involving stakeholders in the generation, revision, and validation of the 
learning outcomes created a sense of commitment and ownership. This involvement also 
generated an intellectual curiosity among the participants about assessment that energized the 
entire assessment process. The subsequent work of the MIB in generating additional scenarios, 
testing assessment tools and scoring rubrics, and evaluating the results, led to their trust that 
assessment would produce results meaningful to industry and ultimately to their advocacy for 
the Coalition and its reform efforts. 

The infusion of industry into the process led to real understanding on the part of both faculty 
and industry of the challenges facing engineering education. This type of collaboration is 
necessary for colleges to bridge the accountability gap between higher education the public it 
serves. Immerwahr’s (1999, p. VI) report on how leader perceptions of higher education differ 
perhaps summarizes best what may be the result of not collaborating: 

The major dispute (between business and education leaders) concerns how well colleges and 
universities are administered, whether they are teaching the right things. . . .These 
disagreements send a clear m.essage about the need for dialogue and clarification between 
leaders inside and outside the halls of academia. Higher education leaders are convinced that 
they will need increasing financial support from society at large. But they may have trouble 
getting that support if they cannot convince other leaders. . . that higher education is doing its 
own work effectively. 
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